Display device and display method

ABSTRACT

A first object present in the visual field of a user is detected together with the position of the first object. A virtual image linked with the first object in advance is displayed together with the first object. A second object operable by the user is detected together with the position of the second object. When an overlapping positional relation viewed from the user between the first object and the second object is a first positional relation, a display form of the virtual image is differentiated from a display from in a second positional relation different from the first positional relation. Consequently, it is possible to perform flexible display.

The present application is based on, and claims priority from JPApplication Serial Number 2019-124303, filed Jul. 3, 2019, thedisclosure of which is hereby incorporated by reference herein in itsentirety.

BACKGROUND 1. Technical Field

The present disclosure relates to a technique for displaying a virtualimage in a visual field

2. Related Art

In recent years, various display devices that display a virtual image inthe visual field of a user such as an HMD have been proposed. In suchdisplay devices, a virtual image is linked with an actually existingobject in advance. When the user views this object using, for example,the HMD, an image prepared in advance is superimposed and displayed on apart or the entire object or displayed near the object (see, forexample, Japanese Patent No. 6264855 (Patent Literature 1)).

Such display devices are useful in that, when a surgical operation isperformed, information necessary for the surgical operation can besuperimposed on a human body as an image or, in manufacturing andmaintenance of a machine and an apparatus, information necessary forwork can be superimposed on a component and a product as an image. Insuch display devices, when a virtual image is linked with a real objectand displayed, a desirable display form of another object, with whichthe virtual image is not linked, such as a surgical instrument or a toolor a user's hand or the like and the virtual image has been furtherimproved.

SUMMARY

The present disclosure can be implemented as a display device thatdisplays an image in a visual field of a user who is capable of visuallyrecognizing an outside scene. The display device includes: a firstdetecting section configured to detect a first object present in thevisual field of the user together with a position of the first object; asecond detecting section configured to detect a second object operableby the user together with a position of the second object; and a displaychanging section configured to superimpose and display, on the firstobject, a virtual image linked with the first object in advance and,when an overlapping positional relation viewed from the user between thefirst object and the second object is a first positional relation,differentiate a display form of the virtual image from a display from ina second positional relation different from the first positionalrelation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory diagram illustrating an exterior configurationof an HMD in a first embodiment.

FIG. 2 is a main part plan view illustrating the configuration of anoptical system included in an image display section.

FIG. 3 is an explanatory diagram illustrating a main part configurationof the image display section viewed from a user.

FIG. 4 is a flowchart illustrating an overview of display processing inthe first embodiment.

FIG. 5 is an explanatory diagram illustrating an example in which theuser is viewing a patient' hand as an outside scene.

FIG. 6 is an explanatory diagram illustrating an example of a virtualimage superimposed and displayed on the patient's hand.

FIG. 7 is an explanatory diagram illustrating an example of appearanceat the time when a user's hand comes close to the patient's hand onwhich the virtual image is superimposed.

FIG. 8 is an explanatory diagram illustrating a comparative example ofthe appearance at the time when the user's hand comes close to thepatient's hand on which the virtual image is superimposed.

FIG. 9A is an explanatory diagram illustrating a positional relationbetween a first object and a second object.

FIG. 9B is an explanatory diagram illustrating appearance at the timewhen a virtual image is superimposed on the first object.

FIG. 10 is an explanatory diagram illustrating a first positionalrelation between the first object and the second object.

FIG. 11A is an explanatory diagram illustrating a state in which thesecond object overlapping the virtual image is removed from the virtualimage.

FIG. 11B is an explanatory diagram illustrating appearance at the timewhen the second object is superimposed on the virtual image illustratedin FIG. 11A.

FIG. 12 is an explanatory diagram illustrating a state in which thesecond object is hidden by the virtual image.

FIG. 13 is an explanatory diagram illustrating appearance at the timewhen the virtual image is a wire frame.

FIG. 14 is an explanatory diagram illustrating appearance at the timewhen the virtual image is deformed.

FIG. 15 is an explanatory diagram illustrating appearance at the timewhen the virtual image is displayed thin.

FIG. 16 is a flowchart illustrating an overview of display processing ina second embodiment.

FIG. 17 is an explanatory diagram illustrating an example of appearanceof a virtual image in the second embodiment.

FIG. 18 is an explanatory diagram illustrating an example of appearanceat the time when display of the virtual image in the second embodimentis corrected.

DESCRIPTION OF EXEMPLARY EMBODIMENTS A. First Embodiment

A-1. Overall Configuration of an HMD

FIG. 1 is a diagram illustrating an exterior configuration of an HMD(Heard Mounted Display) 100 in a first embodiment of the presentdisclosure. The HMD 100 is a display device including an image displaysection 20 (a display section) configured to cause a user to visuallyrecognize a virtual image in a state worn on the user's head and acontrol device 70 (a control section) configured to control the imagedisplay section 20. The control device 70 exchanges signals with theimage display section 20 and performs control necessary for causing theimage display section 20 to display an image.

The image display section 20 is a wearing body worn on the user's head.In this embodiment, the image display section 20 has an eyeglass shape.The image display section 20 includes a right display unit 22, a leftdisplay unit 24, a right light guide plate 26, and a left light guideplate 28 in a main body including a right holding section 21, a leftholding section 23, and a front frame 27.

The right holding section 21 and the left holding section 23respectively extend backward from both end portions of the front frame27 and, like temples of eyeglasses, hold the image display section 20 onthe user's head. Of both the end portions of the front frame 27, an endportion located on the right side of the user in the worn state of theimage display section 20 is represented as an end portion ER and an endportion located on the left side of the user in the worn state of theimage display section 20 is represented as an end portion EL. The rightholding section 21 is provided to extend from the end portion ER of thefront frame 27 to a position corresponding to the right temporal regionof the user in the worn state of the image display section 20. The leftholding section 23 is provided to extend from the end portion EL of thefront frame 27 to a position corresponding to the left temporal regionof the user in the worn state of the image display section 20.

The right light guide plate 26 and the left light guide plate 28 areprovided in the front frame 27. The right light guide plate 26 islocated in front of the right eye of the user in the worn state of theimage display section 20 and causes the right eye to visually recognizean image. The left light guide plate 28 is located in front of the lefteye of the user in the worn state of the image display section 20 andcauses the left eye to visually recognize the image.

The front frame 27 has a shape obtained by coupling one end of the rightlight guide plate 26 and one end of the left light guide plate 28. Theposition of the coupling corresponds to the position of the middle ofthe eyebrows of the user in the worn state of the image display section20. In the front frame 27, a nose pad section in contact with the noseof the user in the worn state of the image display section 20 may beprovided in the coupling position of the right light guide plate 26 andthe left light guide plate 28. In this case, the image display section20 can be held on the user's head by the nose pad section and the rightholding section 21 and the left holding section 23. A belt in contactwith the back of the user's head in the worn state of the image displaysection 20 may be coupled to the right holding section 21 and the leftholding section 23. In this case, the image display section 20 can befirmly held on the user's head with the belt.

The right display unit 22 performs image display by the right lightguide plate 26. The right display unit 22 is provided in the rightholding section 21 and located near the right temporal region of theuser in the worn state of the image display section 20. The left displayunit 24 performs image display by the left light guide plate 28. Theleft display unit 24 is provided in the left holding section 23 andlocated near the left temporal region of the user in the worn state ofthe image display section 20.

The right light guide plate 26 and the left light guide plate 28 in thisembodiment are optical sections (for example, prisms or holograms)formed of light transmissive resin or the like and guide image lightsoutput by the right display unit 22 and the left display unit 24 to theeyes of the user. Dimming plates may be provided on the surfaces of theright light guide plate 26 and the left light guide plate 28. Thedimming plates are thin plate-like optical elements having differenttransmittance according to a wavelength region of light and function asso-called wavelength filters. For example, the dimming plates aredisposed to cover the surface (the surface on the opposite side of thesurface opposed to the eyes of the user) of the front frame 27. Byselecting an optical characteristic of the dimming plates asappropriate, it is possible to adjust the transmittance of light in anywavelength region such as visible light, infrared light, and ultravioletlight. It is possible to adjust a light amount of external light madeincident on the right light guide plate 26 and the left light guideplate 28 from the outside and transmitted through the right light guideplate 26 and the left light guide plate 28.

The image display section 20 guides image lights respectively generatedby the right display unit 22 and the left display unit 24 to the rightlight guide plate 26 and the left light guide plate 28 and causes theuser to visually recognize a virtual image with the image lights (thisis referred to as “display an image” as well). When the external lightis transmitted through the right light guide plate 26 and the left lightguide plate 28 from the front of the user and made incident on the eyesof the user, the image lights forming the virtual image and the externallight are made incident on the eyes of the user. Accordingly, thevisibility of the virtual image in the user is affected by the intensityof the external light.

Accordingly, for example, by mounting the dimming plates on the frontframe 27 and selecting or adjusting the optical characteristic of thedimming plates as appropriate, it is possible to adjust easiness invisually recognizing the virtual image. In a typical example, it ispossible to select dimming plates having light transmissivity enough forthe user wearing the HMD 100 to at least visually recognize an outsidescene. When the dimming plates are used, it is possible to expecteffects of protecting the right light guide plate 26 and the left lightguide plate 28 and suppressing damage, soil adhesion, and the like tothe right light guide plate 26 and the left light guide plate 28. Thedimming plates may be detachably attachable to the front frame 27 oreach of the right light guide plate 26 and the left light guide plate28. A plurality of types of dimming plates may be exchangeable anddetachably attachable or the dimming plates may be omitted.

Besides the members relating to the image display explained above, twocameras 61R and 61L, an inner camera 62, an illuminance sensor 65, asix-axis sensor 66, and an indicator 67 are provided in the imagedisplay section 20. The two cameras 61R and 61L are disposed on theupper side of the front frame 27 of the image display section 20. Thetwo cameras 61R and 61L are provided in positions substantiallycorresponding to both the eyes of the user and are capable of performingmeasurement of the distance to a target object by so-called binocularvision. The distance measurement is performed by the control device 70.The cameras 61R and 61L may be provided anywhere in a position where thedistance measurement by the binocular vision can be performed. Thecameras 61R and 61L may be respectively disposed at the end portions ERand EL of the front frame 27. The measurement of the distance to thetarget object can also be realized by, for example, a configuration inwhich the measurement is performed by a monocular camera and an analysisof an image photographed by the monocular camera or a configuration inwhich the measurement is performed by a millimeter wave radar.

The cameras 61R and 61L are digital cameras including imaging elementssuch as CCDs or CMOS and imaging lenses. The cameras 61R and 61L imageat least a part of an outside scene (a real space) in the front sidedirection of the HMD 100, in other words, a visual field directionvisually recognized by the user in the worn state of the image displaysection 20. In other words, the cameras 61R and 61L image a range or adirection overlapping the visual field of the user and image a directionvisually recognized by the user. In this embodiment, the width of anangle of view of the cameras 61R and 61L is set to image the entirevisual field of the user visually recognizable by the user through theright light guide plate 26 and the left light guide plate 28. An opticalsystem capable of setting the width of the angle of view of the cameras61R and 61L as appropriate may be provided.

Like the cameras 61R and 61L, the inner camera 62 is a digital cameraincluding an imaging element such as a CCD or a CMOS and an imaginglens. The inner camera 62 images the inner side direction of the HMD100, in other words, a direction opposed to the user in the worn stateof the image display section 20. The inner camera 62 in this embodimentincludes an inner camera for imaging the right eye of the user and aninner camera for imaging the left eye. In this embodiment, the width ofan angle of view of the inner camera 62 is set in a range in which theentire right eye or left eye of the user can be imaged. The inner camera62 is used to detect the positions of the eyeballs, in particular, thepupils of the user and calculate the direction of the visual line of theuser from the positions of the pupils of both the eyes. It goes withoutsaying that an optical system capable of setting the width of the angleof view may be provided in the inner camera 62. The inner camera 62 maybe used to read an expression or the like of the user by imaging notonly the pupils of the user but also a wider region.

The illuminance sensor 65 is provided at the end portion ER of the frontframe 27 and disposed to receive external light from the front of theuser wearing the image display section 20. The illuminance sensor 65outputs a detection value corresponding to a light reception amount(light reception intensity). The LED indicator 67 is disposed at the endportion ER of the front frame 27. The LED indicator 67 is lit duringexecution of the imaging by the cameras 61R and 61L to inform that theimaging is being performed.

The six-axis sensor 66 is an acceleration sensor and detects movementamounts in X, Y, and Z directions (three axes) of the user's head andtilts (three axes) with respect to the X, Y, and Z directions of theuser's head. In the X, Y, and Z directions, the Z direction is adirection along the gravity direction, the X direction is a directionfrom the back to the front of the user, and the Y direction is adirection from the left to the right of the user. The tilts of the headare angles around axes (an X axis, a Y axis, and a Z axis) in the X, Y,and Z directions. By integrating signals from the six-axis sensor 66, itis possible to learn a movement amount of the user's head from aninitial position and an angle of the user's head.

The image display section 20 is coupled to the control device 70 by aconnection cable 40. The connection cable 40 is drawn out from thedistal end of the left holding section 23 and detachably coupled to, viaa relay connector 46, a connector 77 provided in the control device 70.The connection cable 40 includes a headset 30. The headset 30 includes amicrophone 63 and a right earphone 32 and a left earphone 34 worn on theright and left ears of the user. The headset 30 is coupled to the relayconnector 46 and integrated with the connection cable 40.

A-2. Configuration of the Control Device

The control device 70 includes, as illustrated in FIG. 1, a right-eyedisplay section 75, a left-eye display section 76, a signal input andoutput section 78, and an operation section 79 besides a CPU 71, amemory 72, a display section 73, and a communication section 74, whichare well known. A predetermined OS is incorporated in the control device70. The CPU 71 realizes various functions by executing, under control bythe OS, programs stored in the memory 72. In FIG. 1, examples of therealized functions are illustrated as a first detecting section 81, asecond detecting section 82, a display changing section 83, and the likein the CPU 71.

The display section 73 is a display provided in a housing of the controldevice 70. The display section 73 displays various kinds of informationconcerning display on the image display section 20. A part or all ofthese kinds of information can be changed by operation using theoperation section 79. The communication section 74 is coupled to acommunication station using a 4G or 5G communication network. Therefore,the CPU 71 is capable of accessing a network via the communicationsection 74 and is capable of acquiring information and images from Websites and the like on the network. When acquiring images, information,and the like through the Internet and the like, the user can select, byoperating the operation section 79, files of moving images and imagesthat the user causes the image display section 20 to display.Alternatively, the user can also select various settings concerning theimage display section 20, conditions for using the HMD 100 such asbrightness of images to be displayed and an upper limit of a continuoustime of use, and the like. It goes without saying that, since the usercan cause the image display section 20 itself to display suchinformation, such processing and setting are possible without thedisplay section 73.

The signal input and output section 78 is an interface circuit thatexchanges signals from the other devices excluding the right displayunit 22 and the left display unit 24, that is, the cameras 61R and 61L,the inner camera 62, the illuminance sensor 65, and the indicator 67incorporated in the image display section 20. The CPU 71 can read, viathe signal input and output section 78, captured images of the cameras61R and 61L and the inner camera 62 of the image display section 20 fromthe cameras 61R and 61L and the inner camera 62 and light the indicator67.

The right-eye display section 75 outputs, with the right display unit22, via the right light guide plate 26, an image that the right-eyedisplay section 75 causes the right eye of the user to visuallyrecognize. Similarly, the left-eye display section 76 outputs, with theleft display unit 24, via the left light guide plate 28, an image thatthe left-eye display section 76 causes the left eye of the user tovisually recognize. The CPU 71 calculates a position of an image thatthe CPU 71 causes the user to recognize, calculates a parallax of thebinocular vision such that a virtual image can be seen in the position,and outputs right and left images having the parallax to the rightdisplay unit 22 and the left display unit 24 via the right-eye displaysection 75 and the left-eye display section 76.

An optical configuration for causing the user to recognize an imageusing the right display unit 22 and the left display unit 24 isexplained. FIG. 2 is a main part plan view illustrating theconfiguration of an optical system included in the image display section20. For convenience of explanation, a right eye RE and a left eye LE ofthe user are illustrated in FIG. 2. As illustrated in FIG. 2, the rightdisplay unit 22 and the left display unit 24 are symmetricallyconfigured.

As components for causing the right eye RE to visually recognize avirtual image, the right display unit 22 functioning as a right imagedisplay section includes an OLED (Organic Light Emitting Diode) unit 221and a right optical system 251. The OLED unit 221 emits image light L.The right optical system 251 includes a lens group and guides the imagelight L emitted by the OLED unit 221 to the right light guide plate 26.

The OLED unit 221 includes an OLED panel 223 and an OLED driving circuit225 configured to drive the OLED panel 223. The OLED panel 223 is aself-emission type display panel that emits light with organicelectroluminescence and is configured by light emitting elements thatrespectively emit color lights of R (red), G (green), and B (blue). Onthe OLED panel 223, a plurality of pixels, a unit of which including oneeach of R, G, and B elements is one pixel, are arranged in a matrixshape.

The OLED driving circuit 225 executes selection and energization of thelight emitting elements included in the OLED panel 223 according to asignal sent from the right-eye display section 75 of the control device70 and causes the light emitting elements to emit light. The OLEDdriving circuit 225 is fixed to the rear side of the rear surface, thatis, a light emitting surface of the OLED panel 223 by bonding or thelike. The OLED driving circuit 225 may be configured by, for example, asemiconductor device that drives the OLED panel 223 and mounted on asubstrate fixed to the rear surface of the OLED panel 223. A temperaturesensor 217 explained below is mounted on the substrate. In the OLEDpanel 223, a configuration in which light emitting elements that emitlight in white are arranged in a matrix shape and color filterscorresponding to the colors of R, G, and B are superimposed and arrangedmay be adopted. The OLED panel 223 having a WRGB configuration includinglight emitting elements that emit white (W) light in addition to thelight emitting elements that respectively emit the R, G, and B lightsmay be adopted.

The right optical system 251 includes a collimate lens that collimatesthe image light L emitted from the OLED panel 223 into light beams in aparallel state. The image light L collimated into the light beams in theparallel state by the collimate lens is made incident on the right lightguide plate 26. A plurality of reflection surfaces that reflect theimage light L are formed in an optical path for guiding light on theinside of the right light guide plate 26. The image light L is guided tothe right eye RE side through a plurality of times of reflection on theinside of the right light guide plate 26. A half mirror 261 (areflection surface) located in front of the right eye RE is formed onthe right light guide plate 26. After being reflected on the half mirror261, the image light L is emitted from the right light guide plate 26 tothe right eye RE and forms an image on the retina of the right eye RE tocause the user to visually recognize a virtual image.

As components for causing the left eye LE to visually recognize avirtual image, the left display unit 24 functioning as a left imagedisplay section includes an OLED unit 241 and a left optical system 252.The OLED unit 241 emits the image light L. The left optical system 252includes a lens group and guides the image light L emitted by the OLEDunit 241 to the left light guide plate 28. The OLED unit 241 includes anOLED panel 243 and an OLED driving circuit 245 that drives the OLEDpanel 243. Details of the sections are the same as the details of theOLED unit 221, the OLED panel 223, and the OLED driving circuit 225. Atemperature sensor 239 is mounted on a substrate fixed to the rearsurface of the OLED panel 243. Details of the left optical system 252 isthe same as the details of the right optical system 251.

With the configuration explained above, the HMD 100 can function as asee-through type display device. That is, the image light L reflected onthe half mirror 261 and the external light OL transmitted through theright light guide plate 26 are made incident on the right eye RE of theuser. The image light L reflected on a half mirror 281 and the externallight OL transmitted through the left light guide plate 28 are madeincident on the left eye LE of the user. In this way, the HMD 100superimposes the image light L of the image processed on the inside andthe external light OL and makes the image light L and the external lightOL incident on the eyes of the user. As a result, for the user, anoutside scene (a real world) is seen through the right light guide plate26 and the left light guide plate 28 and the virtual image by the imagelight L is visually recognized as overlapping the outside scene. Thatis, the image display section 20 of the HMD 100 transmits the outsidescene to cause the user to visually recognize the outside scene inaddition to the virtual image.

The half mirror 261 and the half mirror 281 reflect the image lightsrespectively output by the right display unit 22 and the left displayunit 24 and extract images. The right optical system 251 and the rightlight guide plate 26 are collectively referred to as “right light guidesection” as well. The left optical system 252 and the left light guideplate 28 are collectively referred to as “left light guide section” aswell. The configuration of the right light guide section and the leftlight guide section is not limited to the example explained above. Anysystem can be used as long as the right light guide section and the leftlight guide section form a virtual image in front of the eyes of theuser using the image lights. For example, in the right light guidesection and the left light guide section, a diffraction grating may beused or a semi-transmissive reflection film may be used.

FIG. 3 is a diagram illustrating a main part configuration of the imagedisplay section 20 viewed from the user. In FIG. 3, illustration of theconnection cable 40, the right earphone 32, and the left earphone 34 isomitted. In a state illustrated in FIG. 3, the rear sides of the rightlight guide plate 26 and the left light guide plate 28 can be visuallyrecognized. The half mirror 261 for irradiating image light on the righteye RE and the half mirror 281 for irradiating image light on the lefteye LE can be visually recognized as substantially square regions. Theuser visually recognizes an outside scene through the entire right andleft light guide plates 26 and 28 including the half mirrors 261 and 281and visually recognizes rectangular display images in the positions ofthe half mirrors 261 and 281.

The user wearing the HMD 100 having the hardware configuration explainedabove can visually recognize an outside scene through the right lightguide plate 26 and the left light guide plate 28 of the image displaysection 20 and can further view images formed on the panels 223 and 243as a virtual image via the half mirrors 261 and 281. That is, the userof the HMD 100 can superimpose and view the virtual image on a realoutside scene. The virtual image may be an image created by computergraphics as explained below or may be an actually captured image such asan X-ray photograph or a photograph of a component. The “virtual image”is not an image of an object actually present in an outside scene andmeans an image displayed by the image display section 20 to be visuallyrecognizable by the user.

A-3. Image Display Processing

Processing for displaying such a virtual image and appearance in thatcase are explained below. FIG. 4 is a flowchart illustrating processingexecuted by the control device 70. The processing is repeatedly executedwhile a power supply of the HMD 100 is on. In this embodiment, the useris a surgeon, wears the HMD 100, and, as an example, is about to performa surgical operation for a patient's hand. This case is explained below.

When the processing illustrated in FIG. 4 is started, first, the controldevice 70 performs processing for photographing an outside scene withthe cameras 61R and 61L (step S300). The control device 70 capturesimages photographed by the cameras 61R and 61L via the signal input andoutput section 78. The CPU 71 performs processing for analyzing theimages and detecting an object (step S310). These kinds of processingmay be performed using one of the cameras 61R and 61L, that is, using animage photographed by a monocular camera. If the images photographed bythe two cameras 61R and 61L disposed a predetermine distance apart areused, stereoscopic vision is possible. Object detection can beaccurately performed. The object detection is performed for all objectspresent in the outside scene. Therefore, if a plurality of objects arepresent in the outside scene, the plurality of objects are detected.This processing is equivalent to detection of a first object by thefirst detecting section 81 of the CPU 71.

Subsequently, the control device 70 determines whether an objectregistered in advance is present among the detected objects (step S315).If a registered object is absent among the detected objects, the controldevice 70 returns to step S300 and repeats the processing from thephotographing by the cameras 61R and 61L. In this embodiment, it isassumed that a patient's hand is registered in advance as the object.When determining that the patient's hand, which is the object registeredin advance, is present among the objects detected in the imagesphotographed by the cameras 61R and 61L (YES in step S315), the controldevice 70 performs processing for calculating a position of the object(step S320).

FIG. 5 is illustrates a hand OBJ of the patient photographed by thecameras 61R and 61L. When the user's hand OBJ registered in advance isdetected, the control device 70 calculates a position including thedistance to the hand. The distance to the object and the position on thespace of the object can be easily calculated using a parallax of imagesfrom the right and left cameras 61R and 61L.

Subsequently, the control device 70 generates computer graphics CG,which is display content linked with the detected object in advance(step S330). In this embodiment, the control device 70 generatescomputer graphics CG as a virtual image linked with the patient's handbased on an X-ray photograph of the patient's hand registered inadvance. Then, the control device 70 displays the generated computergraphics CG on the image display section 20 (step S340). This example isillustrated in FIG. 6. In FIG. 6, an outside scene viewed by the user,that is, a visual field range of the user is illustrated as a sign VR. Avirtual image superimposed on the visual field range VR is illustratedas a sign VI. In the visual field range VR, the actual patient's handOBJ is seen. The computer graphics CG is displayed to be superimposed onthe patient's hand OBJ. An image of the computer graphics CG is avirtual image of bones of the fingers of the patient. The image isdisplayed in the visual field of the user by outputting signals from theright-eye display section 75 and the left-eye display section 76 to theright display unit 22 and the left display unit 24 of the image displaysection 20. Specifically, images formed on the right and left panels 223and 243 are guided to the front of the pupils of the user via the rightlight guide plate 26 and the left light guide plate 28 and recognized bythe user via the half mirrors 261 and 281.

A display position of the virtual image is changed according to amovement of the user's head. A relative positional relation with thefirst object is kept constant. Such update of the display position ofthe virtual image may be performed by analyzing the images from thecameras 61R and 61L. A signal from the six-axis sensor 66 may beanalyzed to recognize a movement of the user's head. The displayposition may be updated according to the movement of the user's head.

In this state, the user recognizes that the image of the bones issuperimposed and displayed on the patient's hand. A display form at thetime when the surgeon, who is the user, brings a hand holding a surgicalknife close to the patient's hand is explained. This state isillustrated in FIG. 7. At this time, the control device 70 sends aninstruction to the cameras 61R and 61L and causes the cameras 61R and61L to photograph the outside scene again (step S350). Then, the controldevice 70 performs processing for detecting a second object includingthe user's hand (step S360). The second object is an object that can beoperated by the user. The second object may be the user's hand, may bean object such as a surgical knife or a pointing rod, or may be the handand a thing gripped by the hand integrally grasped as an object. In anexample illustrated in FIG. 7, the second object is the user's hand andthe surgical knife gripped by the hand. Since the second object isregistered in the memory 72 as the hand and the surgical knife grippedby the hand, even if a way of holding the surgical knife is different,the control device can integrally recognize the hand and the surgicalknife. Specifically, a color range of the hand and the surgical knife isregistered in advance. A continuous range of a region included in thecolor range is recognized as the second object. Such recognition can beeasily realized by labeling or the like. In FIG. 7, the range recognizedas the second object is illustrated as a sign SC.

With the hand detection processing (step S360), the control device 70determines whether the hand is successfully detected (step S362). If thesecond object including the hand is not successfully detected, thecontrol device 70 returns to step S300 and repeats the processingexplained above. If the second object including the hand is successfullydetected (YES in step S362), subsequently, the control device 70determines whether the user's hand and the surgical knife, which are thesecond object, overlap the patient's hand, which is the object OBJserving as the first object, detected earlier (step S365). If the user'shand and the surgical knife and the patient's hand do not overlap, thecontrol device 70 returns to step S300 and repeats the processingexplained above. The user's hand and the surgical knife and thepatient's hand overlapping is equivalent to the first object and thesecond object being in a first positional relation. The user's hand andthe surgical knife and the patient's hand not overlapping is equivalentto the first object and the second object being in a second positionalrelation.

When determining that the first object and the second object overlap(YES in step S365), the control device 70 performs processing forcorrecting the computer graphics CG (step 370) and subsequently performsprocessing for displaying the corrected computer graphics CG on theimage display section (step S380). After the series of processing ends,the control device 70 once ends the processing illustrated in FIG. 4.However, the processing illustrated in FIG. 4 is repeated at apredetermined interval as explained above. Therefore, the processing isrepeated from step S300 again. The detection of the patient's hand,which is the first object, the specifying of the position of thepatient's hand, the detection of the user's hand and the surgical knife,which are the second object, the specifying of the positions of theuser's hand and the surgical knife, the generation of the computergraphics CG, and the correction and the display according to the overlapare continuously performed.

The determination of the overlap of the patient's hand, which is thefirst object, and the user's hand and the surgical knife, which are thesecond object, and the correction and the display of the computergraphics CG are explained. As illustrated in FIG. 3, if the processingin steps S365 to S380 illustrated in FIG. 4 is not performed, whenviewed from the user, the half mirrors 261 and 281 of the image displaysection 20 are present and the computer graphics CG is displayed on thenearer side than the outside scene. Therefore, the computer graphics CGis superimposed and displayed on an object present as the outside scene.This state is illustrated in FIG. 8.

Even if the user's hand and the surgical knife come close to thepatient's hand and are in a positional relation in which the user's handand the surgical knife overlap the patient's hand when viewed from theuser, the computer graphics CG superimposed and displayed on thepatient's hand is superimposed and displayed on the entire outsidescene. That is, since the computer graphics CG is superimposed anddisplayed on the user's hand and the surgical knife as well, apositional relation between the patient's hand and the computer graphicsCG and the user's hand and the surgical knife viewed from the user couldbe different from originally assumed appearance. An X-ray video of thebones of the patient's hand displayed as the computer graphics CGdisplays a state of the bones originally invisible from the outside ofthe patient's hand and assists a surgical operation. A presence positionof the X-ray video is assumed to be the inside of the patient's hand.Nevertheless, as illustrated in FIG. 8, since the computer graphics CGis displayed on the half mirrors 261 and 281 of the image displaysection 20, the computer graphics CG is displayed to overlap the user'shand as well.

On the other hand, in this embodiment, the control device 70 determinesthe overlap of the patient's hand, which is the first object, and theuser's hand, the surgical knife, and the like, which are the secondobject, (step S365) and, when the patient's hand and the user's hand,the surgical knife, and the like overlap, corrects the computer graphicsCG (step S370) and displays the computer graphics CG (step S380). As aresult, instead of the display illustrated in FIG. 8, as illustrated inFIG. 7, the computer graphics CG overlapping the second object presenton the near side of the overlap is corrected. The computer graphics CGin the portion of the user's hand and the surgical knife, which are thesecond object, is deleted. Therefore, when viewed from the user, theuser's hand and the surgical knife are displayed on the nearest side,the patient's hand, which is the first object, and the X-ray photographof the bones, which is a virtual image superimposed on the patient'shand, are displayed behind the user's hand and the surgical knife. Inthe first embodiment, in which way, display without discomfort isrealized when viewed from the user.

A-4. Other Display Forms of the Virtual Image

Various measures are possible other than the embodiment to change thedisplay form of the virtual image from the overlapping positionalrelation between the first object and the second object operable by theuser. Forms of display by the measures are explained with reference toexamples illustrated in FIGS. 9A and 9B and subsequent figures. FIG. 9Aillustrates a state in which a can OB1 of drinking water, which is thefirst object, and a user's hand IG1, which is the second object, areapart. At this time, the HMD 100 recognizes the first object from imagescaptured by the cameras 61R and 61L. As illustrated in FIG. 9B, the HMD100 superimposes and displays a virtual image AR1 on the can OB1.

It is assumed that, from this state, the user's hand IG1 comes close tothe can OB1, which is the first object, and, as illustrated in FIG. 10,the user's hand IG1 and the can OB1 overlap and the hand IG1 is presenton the near side. At this time, as in the first embodiment, asillustrated in FIG. 11A, the virtual image AR1 is corrected to a virtualimage AR2 shaved off in the shape of the hand IG1, which is the secondobject. In this way, a part of the virtual image AR1 is hidden by thehand IG1, which is the second object. It is seen as if the virtual imageAR1 have changed to the virtual image AR2. As a result, the user's handIG1 is visually recognized as if the user's hand IG1 is present on thevirtual image AR1.

On the other hand, if such correction of the virtual image AR1 is notperformed, as illustrated in FIG. 12, the user's hand IG1 is hidden bythe virtual image AR1. It is seen as if the user's hand IG1 is presenton the other side of the first object OB1. Therefore, by correcting thevirtual image according to the overlap of the two objects, in this case,it is possible to realize display corresponding to actual overlap of theobjects.

In contrast, if the user's hand IG1 is present on the other side of thecan OB1, which is the first object, the display illustrated in FIG. 12is display without discomfort when viewed from the user. In this case,it is unnecessary to change the display form of the virtual image AR1.On the other hand, as another form, it is also possible that displayillustrated in FIG. 11B is performed if the user's hand IG1 is presenton the other side of the can OB1, which is the first object, and displayillustrated in FIG. 12 is performed if the user's hand IG1 is present onthe near side of the can OB1, which is the first object. In this case,the user can enjoy an experience of appearance different from a reality,so to speak, an experience in a different dimension. Therefore, when theoverlapping positional relation viewed from the user between the canOB1, which is the first object, and the hand IG1, which is the secondobject, is the first positional relation, by differentiating the displayform of the virtual image from a display form of the virtual image atthe time when the positional relation is the second positional relationdifferent from the first positional relation, it is possible to realizeone of natural display (without discomfort or with less discomfort)viewed from the user and display with which the user can experienceappearance different from the reality. The former is suitable forsupporting work such as a surgical operation and repair in which avirtual image is used and the latter can be used for a game and thelike.

Another form of differentiating the display form of the virtual image isexplained. For example, when the second object overlaps the firstobject, as illustrated in FIG. 13, the virtual image may be an image AR3of a wire frame. If the virtual image is the wire frame, the firstobject can be visually recognized. Even if the wire frame is displayedfurther on the user side than the user's hand IG1, which is the secondobject, display without discomfort is performed. At this time, the wireframe in a position overlapping the user's hand, which is the secondobject, may be erased.

Alternatively, as illustrated in FIG. 14, when the user's hand IG1,which is the second object, comes close to the virtual image, thevirtual image may be deformed and displayed as an image AR4 notoverlapping the second object. In this case, when the virtual image isdeformed, the first object OB1 present behind the virtual image is seen.Display with less discomfort is performed when viewed from the user.

Further, as illustrated in FIG. 15, when the user's hand, which is thesecond object, comes close to the virtual image, the display of thevirtual image may be thinned and displayed as an image AR5 through whichthe second object is seen. In this case, since the virtual image isdisplayed thin, the first object OB1 present behind the virtual image isalso seen through and the user's hand IG1, which is the second object,is also seen through. Therefore, display with less discomfort isperformed when viewed from the user.

All of the display examples illustrated in FIGS. 13 to 15 are displaywithout discomfort or less discomfort when viewed from the user.Conversely, when the first object and the second object do not overlap,a wire frame may be displayed and, when the first object and the secondobject overlap, solid display illustrated in FIG. 9B may be performed.In the examples illustrated in FIGS. 14 and 15, similarly, the displayform at the time when the first object and the second object do notoverlap and the display form at the time when the first object and thesecond object overlap may be reversed. In this case, for the user,display with discomfort is performed. The user can enjoy a visualexperience in a different dimension. Such display may be performed in agame and the like.

The HMD 100 can also be used for assembly, repair, and the like of aproduct. In such a case, a plurality of components of a product beingassembled or a product to be repaired are seen in an outside scene. Avirtual image is superimposed on the plurality of components to guideassembly work or guide repair. Even in such a case, when a user's hand,a tool, or the like comes close to the virtual image, a positionalrelation between a component, which is the first object, and the user'shand or the tool, which is the second object, is determined. Bydifferentiating a display form of the virtual image in a firstpositional relation in which the component and the user's hand or thetool overlap and a second positional relation in which the component andthe user's hand or the tool do not overlap, it is possible to reducediscomfort in appearance viewed from the user.

B. Second Embodiment

A second embodiment is explained. The HMD 100 in the second embodimenthas the same hardware configuration as the hardware configuration of theHMD 100 in the first embodiment. Only processing performed by the HMD100 in the second embodiment is different from the processing performedby the HMD 100 in the first embodiment. In the HMD 100 in the secondembodiment, a part of processing performed by the control device 70 isdifferent from the processing illustrated in FIG. 4. The control device70 in the second embodiment executes processing illustrated in FIG. 16.This processing is the same as the processing illustrated in FIG. 4 insteps S300 to S362. However, as illustrated in FIG. 17, an outside scenephotographed in step S300 includes a shelf CH and a plurality ofcomponents equivalent to a first object placed on the shelf. The displaycontent generated in step S330, that is, the virtual image linked withthe first object is, in the second embodiment, an explanatory noteconcerning a component. The computer graphics CG of displayedexplanatory notes generated in step S330 is illustrated as signs AR11and AR12 in FIG. 17.

The control device 70 performs the processing in steps S300 to S360 andthen determines whether the user's hand IG1 is detected (step S362).When detecting the user's hand IG1, the control device 70 determineswhether the computer graphics CG, which is the virtual image, and theuser's hand, which is the second object, are in a first positionalrelation, that is, close to each other at a predetermined distance orless (step S368). The second embodiment is different from the firstembodiment in that a target of determination of a positional relationwith the user's hand, which is the second object, is not the firstobject and is the computer graphics CG, which is the virtual image.

If the computer graphics CG and the user's hand are not close to eachother at the predetermined distance or less, the control device 70returns to step S300 and repeats the processing. When determining thatthe user's hand is close to the computer graphics CG at thepredetermined distance or less and the user's hand and the computergraphics CG are about to overlap (YES in step S368), the control device70 corrects a display position of the computer graphics CG (step S375)and displays the computer graphics CG in the corrected display positionusing the image display section 20 (step S380). After the processingexplained above, the control device 70 once ends this processingroutine.

An example of display in the second embodiment is explained. FIG. 17 isan example of an outside scene that the user is viewing in the secondembodiment. In this example, several components are placed on the shelfCH. Among the components, a component OB11 and a component OB12 arefirst objects registered in advance. If the registered first objects arepresent in an outside scene imaged by the cameras 61R and 61L (stepsS300 to S315), the control device 70 recognizes the first objects andgenerates the virtual images AR11 and AR12, that is, texts concerningthe components OB11 and OB12 as the computer graphics CG and displaysthe virtual images AR11 and AR12 near the components OB11 and OB12,which are the first objects (steps S320 to S340). FIG. 17 is a displayexample in the case in which the user's hand IG1 is present in a placefarther than a predetermined distance from the virtual images AR11 andAR12.

When the user stretches the hand to the shelf CH and the hand IG1 comesclose to the virtual images AR11 and AR12 in this state, the controldevice 70 detects the user's hand IG1, which is the second object,calculates a position of the user's hand IG1, and detects whether theuser's hand IG1 is close to the virtual images AR11 and AR12 at thepredetermined distance or less (steps S350 to S368). When determiningthat the user's hand IG1 is close to the virtual images AR11 and AR12 atthe predetermined distance or less, the control device 70 correctsdisplay positions of the virtual images AR11 and AR12 (step S375) andchanges display positions of the virtual images AR11 and AR12. Thisstate is illustrated as FIG. 18. “Close to” in this case means closenessof overlap of the user's hand IG1 and the virtual images AR11 and AR12viewed from the user and, therefore, does not mean a three-dimensionaldistance and means a distance in a plan view from the user. Accordingly,the cameras are capable of also determining “whether the user's hand IG1is close to the virtual images AR11 and AR12” based on an image of amonocular camera.

In the second embodiment, in this way, when the hand of the user usingthe HMD 100 comes close to the images AR11 and AR12 virtually displayedby the HMD 100, the display positions of the images AR11 and AR12 arechanged in advance to prevent the user's hand IG1 from overlapping thecomputer graphics CG. Therefore, it is possible to avoid a situation inwhich the user's hand IG1 is hidden by the virtual images AR11 and AR12and display with discomfort is performed. In a display exampleillustrated in FIG. 18, the virtual image AR11 displayed for thecomponent OB11 is not always close to the user's hand IG1. However,according to correction of a display position of the virtual image AR12,which is likely to come close to and overlap the user's hand IG1, adisplay position of the virtual image AR11 is corrected to prevent thevirtual image AR11 from overlapping the virtual image AR12. It goeswithout saying that the display position of only the virtual image AR12,to which the user's hand IG1 is close at the predetermined distance orless, may be corrected to prevent the virtual image AR12 fromoverlapping not only the user's hand IG1 but also the other virtualimage AR11.

In the embodiment explained above, the virtual images AR11 and AR12 arethe explanatory notes for the component OB11 and the like. However,virtual images do not need to be limited to such explanatory notes andmay be images that apparently expand the first objects. For example, afirst object registered in advance may be a specific individual andvirtual images may be a flag, a sword, or the like held by the specificindividual. When a user's hand comes close to the flag and the sword,which are the virtual images, display positions of the virtual imagesmay be corrected to be away from the hand. Such correction of thedisplay positions can be used in a game and the like. When the virtualimages and the second object are in the first positional relation, thedisplay positions of the virtual images only have to be corrected to bedifferent from display positions at the time when the virtual images andthe second object are in the second positional direction. The displaypositions may be corrected to bring the virtual images close to thesecond object. In this case, when the second object comes close to thevirtual images, the display positions of the virtual images are changedsuch that the virtual images adhere to the second object.

C. Other Embodiments

In the embodiments explained above, the position of the object iscalculated using the images from the cameras 61R and 61L. However,instead of the cameras 61R and 61L, the position and the distance of theobject may be calculated, or derived, using other sensors. For instance,when a tag or the like is embedded in the object, the position and thedistance of the object may be calculated using the strength of a radiowave from the tag. If it is known in advance that the object is presentin a predetermined place, a GSNN receiving device may be provided in thecontrol device 70. The position and the distance of the object may becalculated as differences between the latitude, the longitude, and thealtitude of the HMD 100 and the latitude, the longitude, and thealtitude of the object, the position of which is known in advance.Alternatively, a distance measurement sensor, which detects the distanceto a measurement target object located in a measurement direction set inadvance, may be provided in the HMD 100 to measure the distances to thefirst object and the second object. The distance measurement sensor canbe configured by, for example, a light emitting section such as an LEDor a laser diode and a light receiving section configured to receivereflected light of light emitted from a light source and reflected onthe measurement target object. In this case, a distance is calculated bytriangulation processing or distance measurement processing based on atime difference. The distance measurement sensor may be configured by,for example, a transmitting section configured to emit ultrasound and areceiving section configured to receive the ultrasound reflected on themeasurement target object. In this case, a distance is calculated bydistance measurement processing based on a time difference.

Embodiments explained below are also possible besides the variousembodiments explained above.

(1) As one aspect, it is possible to assume a display device thatdisplays an image in a visual field of a user who is capable of visuallyrecognizing an outside scene, the display device including: a firstdetecting section configured to detect a first object present in thevisual field of the user together with a position of the first object; asecond detecting section configured to detect a second object operableby the user together with a position of the second object; and a displaychanging section configured to superimpose and display, on the firstobject, a virtual image linked with the first object in advance and,when an overlapping positional relation viewed from the user between thefirst object and the second object is a first positional relation,differentiate a display form of the virtual image from a display from ina second positional relation different from the first positionalrelation.

Consequently, it is possible to differentiate the display form of thevirtual image according to whether the overlapping positional relationbetween the first object and the second object is the first positionalrelation or the second positional relation. Therefore, it is possible toflexibly display the first object, the second object, and the virtualimage according to the overlapping positional relation between the firstobject and the second object. The display form of the virtual image inthe first positional relation and the display form of the virtual imagein the second positional relation only have to be different. Forexample, the first positional relation may be a positional relation atthe time when the second object is present further on the user side thanthe first object. The second positional relation may be a positionalrelation at the time when the second object is not present further onthe user side than the first object. In this case, in the firstpositional relation, as the display forms, the second object may bedisplayed as being present further on the user side than the virtualimage and, in the second positional relation, the virtual image may bedisplayed as being present further on the user side than the secondobject. However, the first and second positional relations and thedisplay forms may be reversed. The former is close to natural displaywhen viewed from the user. However, an experience of appearancedifferent from a reality, so to speak, an experience in a differentdimension may be provided to the user by the latter display. Thisdisclosed aspect can also be implemented as a display method.

(2) In such a display device, the first positional relation may be apositional relation in which the second object is present further on theuser side than the first object and at least a part of the second objectoverlaps the first object when viewed from the user. Consequently, it ispossible to detect a positional relation in which the second object ispresent further on the near side than the first object when viewed fromthe user and display the virtual image according to the detectedpositional relation. In this case as well, association between the firstand second positional relations and the display form can be variouslyset. For example, the display form may be changed between when at leasta part of the second object overlaps the first object when viewed fromthe user and when a part of the second object does not overlap the firstobject when viewed from the user. The display form may be a form closeto real appearance when viewed from the user or may be appearancedifferent from a reality.

(3) In such a display device, the display form in the first positionalrelation may be a form in which the virtual display is performedavoiding at least the part of the second object. Consequently, since thevirtual image is displayed to avoid at least the part of the secondobject overlapping the first object on the user side, it is possible toeasily recognize that at least the part of the second object is presentfurther on the user side than the virtual image and the first object.

(4) In such a display device, the display form in the first positionalrelation may be a form in which at least the part of the second objectis transmitted through the virtual display and displayed to be able tobe visually recognized. Consequently, when the second object overlapsthe first object, since at least the part of the second object can bevisually recognized through the virtual display, it is possible toeasily recognize that at least the part of the second object is presentfurther on the user side than the virtual image and the first object.When at least the part of the second object is “transmitted” through thevirtual display and displayed, the virtual display may be displayed thinto enable the second object to be seen through the virtual display. Thevirtual display may be formed as a wire frame or the like to enable thesecond object to be seen.

(5) Another aspect of the present disclosure may be a display devicethat displays an image in a visual field of a user who is capable ofvisually recognizing an outside scene, the display device including: afirst detecting section configured to detect a first object present inthe visual field of the user together with a position of the firstobject; a second detecting section configured to detect a second objectoperable by the user together with a position of the second object; anda display changing section configured to display, with respect to thefirst object, in a predetermined display position, a virtual imagelinked with the first object in advance and, when a positional relationviewed from the user between the display position where the virtualimage is displayed and the second object is a first positional relation,differentiate the display position of the virtual image from a displayposition in a second positional relation different from the firstpositional relation.

Consequently, it is possible to differentiate the display position ofthe virtual image according to whether the positional relation viewedfrom the user between the display position of the virtual image and thesecond object is the first positional relation or the second positionalrelation. Therefore, it is possible to flexibly change the displayposition of the virtual image according to the difference in thepositional relation between the display position of the virtual imageand the second object. The display position of the virtual image in thefirst positional relation and the display position of the virtual imagein the second positional relation only have to be different. Forexample, the first positional relation may be a positional relation atthe time when the second object is away from the display position of thevirtual image when viewed from the user. The second positional relationmay be a positional relation at the time when the second object comesclose to or overlap the display position of the virtual image. At thistime, in the first positional relation, the virtual image may bedisplayed in a predetermined position with respect to the first objectand, in the second positional relation, the virtual image may bedisplayed in a position where the virtual image does not interfere withthe second object. It goes without saying that the first and secondpositional relations and the display positions may be reversed. Theformer is display in which the second object and the virtual image donot interfere when viewed from the user. However, by displaying thevirtual image in a display position where the second object and thevirtual image interfere, an experience of appearance seen as if thevirtual image clings to the second object operated by the user may beprovided to the user. This disclosed aspect can also be implemented as adisplay method.

(6) In such a display device, the second detecting section may detect,as the second object, at least one of a part of a human body and anobject registered in advance operated by a person. Consequently, it ispossible to detect a hand or a foot of the user or fingers of the handor the foot and treat the hand or the foot or the fingers as the secondobject. It is possible to appropriately display a pointing rod, acookware such as a ladle or a spatula, a writing instrument such as apencil or a brush, a tool such as a screwdriver, or the like held by thehand or the foot of the user or the fingers of the hand or the foot andthe virtual image. The human body may be a human body of a person otherthan the user of the display device. The human body can be easilyrecognized by a color including a hue of skin and the area of the color,a marker stuck to the human body, or the like. A person other than theuser may operate the object. In this case, the object only has to beregistered in advance by a marker, a tag, RFID, or the like.

(7) The display device may be a head-mounted display device.Consequently, it is possible to move the virtual image according to achange of the user's head, that is, the visual field and cause thedisplay device to naturally display, in the outside scene, the virtualimage linked with the first object in the outside scene. The displaydevice is not limited to the head-mounted display device and may be ahand-held display device, a monocular display device, or the like if thedisplay device can display the virtual image over the outside scene. Thedisplay device may be a display device that displays the virtual imageon a windshield or the like in a vehicle such as an automobile or aship. In this case, the second object may be a part of a vehicle body ora ship hull. If the vehicle is a heavy machine such as a hook-and-laddertruck or a power shovel, the second object may be an operable structuresuch as a ladder or a shovel.

(8) In the embodiments, a part of the components realized by hardwaremay be replaced with software. At least a part of the componentsrealized by software can also be realized by discrete hardware circuitcomponents. When a part or all of the functions of the presentdisclosure are realized by software, the software (a computer program)can be provided in a form stored in a computer-readable recordingmedium. The “computer-readable recording medium” is not limited to aportable recording medium such as a flexible disk or a CD-ROM andincludes various internal storage devices in a computer such as a RAMand a ROM and external storage devices fixed to the computer such as ahard disk. That is, the “computer-readable recording medium” has a broadmeaning including any recording medium that can record a data packet nottemporarily but fixedly.

The present disclosure is not limited to the embodiments explained aboveand can be realized in various configurations without departing from thegist of the present disclosure. For example, the technical features inthe embodiments corresponding to the technical features in the aspectsdescribed in the summary can be substituted or combined as appropriatein order to solve a part or all of the problems described above orachieve a part of all of the effects described above. Unless thetechnical features are explained as essential technical features in thisspecification, the technical features can be deleted as appropriate.

What is claimed is:
 1. A display device comprising: a display thatdisplays an image in a visual field of a user; a camera; and one or moreprocessors or hardware circuits configured to: detect, using the camera,a first object present in the visual field of the user together with aposition of the first object; detect, using the camera, a second objectoperable by the user together with a position of the second object; andcause the display to superimpose and display, on the first object, avirtual image linked with the first object in advance and, when anoverlapping positional relation viewed from the user between the firstobject and the second object is a first positional relation,differentiate a display form of the virtual image from a display form ina second positional relation different from the first positionalrelation.
 2. The display device according to claim 1, wherein the firstpositional relation is a positional relation in which the second objectis present further on the user side than the first object and at least apart of the second object overlaps the first object when viewed from theuser.
 3. The display device according to claim 2, wherein the displayform in the first positional relation is a form in which the virtualdisplay is performed avoiding at least the part of the second object. 4.The display device according to claim 2, wherein the display form in thefirst positional relation is a form in which at least the part of thesecond object is transmitted through the virtual display and displayedto be able to be visually recognized.
 5. A display device comprising: adisplay that displays an image in a visual field of a user; a camera;and one or more processors or hardware circuits configured to: detect,using the camera, a first object present in the visual field of the usertogether with a position of the first object; detect, using the camera,a second object operable by the user together with a position of thesecond object; and cause the display to display, with respect to thefirst object, in a predetermined display position, a virtual imagelinked with the first object in advance and, when a positional relationviewed from the user between the display position where the virtualimage is displayed and the second object is a first positional relation,differentiate the display position of the virtual image from a displayposition in a second positional relation different from the firstpositional relation.
 6. The display device according to claim 1, whereinthe camera detects, as the second object, at least one of a part of ahuman body and an object registered in advance operated by a person. 7.The display device according to claim 1, wherein the display device is ahead-mounted display device.
 8. A method for displaying an image in avisual field of a user, the display method comprising: detecting a firstobject present in the visual field of the user together with a positionof the first object; displaying a virtual image linked with the firstobject together with the first object; detecting a second objectoperable by the user together with a position of the second object; andwhen an overlapping positional relation viewed from the user between thefirst object and the second object is a first positional relation,differentiating a display form of the virtual image from a display formin a second positional relation different from the first positionalrelation.
 9. A method for displaying an image in a visual field of auser, the display method comprising: detecting a first object present inthe visual field of the user together with a position of the firstobject; displaying a virtual image linked with the first object in apredetermined position with respect to the first object; detecting asecond object operable by the user together with a position of thesecond object; and when a positional relation viewed from the userbetween the display position where the virtual image is displayed andthe second object is a first positional relation, differentiating thedisplay position of the virtual image from a display position in asecond positional relation different from the first positional relation.